Policy Driven Replication

From the inception of digital storage, ensuring that data is not lost due to user error, malicious acts, and hardware failure has always been, and still remains, a challenging open problem. This problem is exacerbated by the exponential increase in storage capacity, the proliferation of new digital media, and our growing reliance on digital storage. Today, a typical user stores financial and medical records, music and movie libraries, photo albums, etc, the loss of some of which can be catastrophic. The advent of large robust networks has made it possible to replicate data on remote hosts to protect data from loss. Unfortunately, the growth of network bandwidth is far outstripped by both the growth of storage capacity and our ability to fill it. Thus, most replication systems that uniformly replicate all the data are incapable of protecting the ever increasing amount of data. One important observation is that not all data is created equal. Data such as commercial music and movie libraries can be, given time, rebuilt. Data such as personal, health, and financial records, are much more difficult to reconstruct. Since resources such as network bandwidth are limited, they should be used to protect the important data. In this thesis we propose a Policy Driven Replication (PDR) system that prioritizes data replication according to user-defined policies that specify what data is to be protected, from what failures, and to what extent. By prioritizing what data is replicated, the system conserves limited resources and protects high-priority data from high-probability failures. PDR is a userlevel process that hooks into the file system. It is notified of file creation and modification events, and replicates the data to the hosts specified in the file’s policy. In addition, the replica nodes specified in the policy are monitored for liveliness to ensure the policy is followed. PDR provides a model to describe replica nodes and a generic plug-in interface that facilitates the creation of appropriate user interfaces to manage replication policies and to translate these policies into a set of replica nodes. Replica node selection is sensitive to the system topology so that hotspots and message storms are not created.